1. Field of the Invention
Embodiments of the present invention relate generally to quality assurance techniques and more specifically to a method and system for automatically verifying the quality of multimedia rendering.
2. Description of the Related Art
Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
The robustness of a multimedia system depends in part on how rigorously and extensively the system is tested. However, as the multimedia system and the applications running on the system become increasingly complex, verifying these complex system and applications in a comprehensive and yet timely manner becomes more and more challenging. To illustrate, FIG. 1A is a simplified diagram of a conventional verification process for a graphics system. In this process, a human operator 110 visually inspects the image outputs of a computing device 100 and a computing device 102 to try to detect any image corruption. The hardware of this verification process is kept constant, so that the graphics processing unit (“GPU”) 106 and the GPU 126 are the same. Some of the software components, such as the application 102 and the application 122, are also kept constant, but the other components, such as the baseline driver 104 and the test driver 124 are varied. Because of the reliance placed on the human operator 110, this process may be slow in delivering verification results, and the verification results are prone to human errors.
Although some prior art attempts have been made to automate the verification process discussed above, such efforts still fall short, especially for handling highly interactive multimedia applications, such as games. FIG. 1B is a simplified diagram illustrating a conventional automated verification process for a graphics system. Here, the process involves a single computing device 130 running two sets of testing procedures at two different points in time, namely, time1 and time2, without any intervention of a human operator. The first run of testing, denoted as run (time1), involves a baseline driver 134, and the second run of testing, denoted as run (time2), involves a test driver 144. The computing device 130 first stores the verification results from run (time1) in a storage device 138 and then retrieves the stored data to compare with the results from run (time2) in a comparison operation 146. Unlike the process shown in FIG. 1A, either the GPU 136 or another processing unit (not shown in FIG. 1A) in the computing device 130 performs the comparison operation 146.
Despite the automation, there are still several drawbacks associated with this verification process. One, due to the limited capacity of the storage device 138, only a limited amount of verification results generated by the process can be stored and retrieved for comparison. Consequently, instead of verifying an entire graphics application, only a few representative frames of data from the graphics application are tested. This lack of extensive testing of the graphics application renders the application less stable. Two, the automated verification process is unable to conduct multiple test runs, such as run (time1) and run (time2), under identical testing conditions and potentially leading to meaningless verifications results. For instance, suppose a newly developed test driver 144 is to be tested against the baseline driver 134 on how a ball 152 bounces along a path 154 in a display screen 150 shown in FIG. 1C. Suppose further that the bouncing pattern of the ball 152 is generated according to a time-based model. So, even if the path 154 stays constant in run (time1) and run (time2), any change in the testing conditions between the two runs may result in displaying the ball 152 at a position 156 in a particular frame in run (time1) and displaying the ball 152 at a completely different position, such as a position 158, in the same frame in run (time2). As has been demonstrated, performing the comparison operation 146 on the aforementioned two frames from the two test runs yields little to no useful information.
Moreover, even if the testing conditions can be kept constant between test runs, the test runs can still generate completely unrelated output data. For example, suppose the test driver 144 is to be tested against the baseline driver 134 on displaying the explosion of the ball 152 in the display screen 150. If the debris pieces from the explosion are designed to be randomly generated, then having the same set of pieces in run (time1) and run (time2) to compare is nearly impossible and again leading to potentially meaningless verification results.
As the foregoing illustrates, what is needed in the art is a verification process that is capable of extensively and efficiently verifying data generated by multimedia applications and addressing at least the shortcomings of the prior art approaches set forth above.